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ABSTRACT 
The intention of this paper is to provide an overview on the subject of Distributed shared memory. Distributed shared memory (DSM) systems represent a 
successful hybrid of two parallel computer classes: shared memory multiprocessors and distributed computer systems. They provide the shared memory 
abstraction in systems with physically distributed memories, and consequently combine the advantages of both approaches. Because of that, the concept of 
distributed shared memory is recognized as one of the most attractive approaches for building large-scale, high-performance multiprocessor systems. The 
increasing importance of this subject imposes the need for its thorough understanding. Problems in this area logically resemble the cache coherence-related 
problems in shared memory multiprocessors. 
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1. INTRODUCTION What 

Traditionally, shared memory and message passing have been the two ° The distributed shared memory {DSM} implements the shared memory 
programming models for interprocess communication and synchronization model in distributed systems, which have ne physical shared memory 

in computations performed on a distributed system. Message passing has * The shared memory model provides a virtual address space shared 
been the preferred way of handling interprocess communication in loosely- between all nodes . re ene 

coupled systems, because the computers forming a distributed system donot * The overcome the high cost of communication in distributed systems, 
share physical memory. The message passing model is characterized by data DSM systems move data to the location of access 

movement among cooperating processes as they communicate and 
synchronize by sending and receiving messages. In contrast, tightly-coupled 
processors primarily use shared memory model since it provides direct 
support for data sharing. 


How 
e Data moves between main memory and secondary memory (within a 
node) and between main memories of different nodes 
e Each data object is owned by a node 
e Initial owner is the node that created object 
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e Ownership can change as object moves from node to node 

e When a process accesses data in the shared address space, the mapping 
manager maps shared memory address to physical memory (local or 
remote). 


2. ALGORITHMS FOR IMPLEMENTING DSM 

Concerns 
e How to keep track of the location of remote data 
e How to minimize communication overhead when accessing remote data 
e How to access concurrently remote data at several nodes 


2.1. The Central Server Algorithm 
Central server maintains all shared data 
e Read request: returns data item 
e Write request: updates data and returns acknowledgement message 


Implementation 
e A timeout is used to resend a request if acknowledgment fails 
e Associated sequence numbers can be used to detect duplicate write 
requests 
e If an application's request to access shared data fails repeatedly, a failure 
condition is sent to the application 


Issues 
e Performance and reliability 


Possible solutions 
e Partition shared data between several servers 
e Use a mapping function to distribute/locate data 


2.2. The Migration Algorithm 
Operation 
e Ship (migrate) entire data object (page, block) containing data item to 
requesting location 
e Allow only one node to access a shared data at a time 


Advantages 
e Takes advantage of the locality of reference 
e DSM can be integrated with VM at each node 


Make DSM page multiple of VM page size 
e A locally held shared memory can be mapped into the VM page address 
space 
e If page not local, fault-handler migrates page and removes it from 
address space at remote node 


To locate a remote data object: 
e Use a location server 
e Maintain hints at each node 
e Broadcast query 


Issues 
e Only one node can access a data object at a time 
e Thrashing can occur: to minimize it, set minimum time data object 
resides at a node 


2.3. The Read-Replication Algorithm 
Replicates data objects to multiple nodes 
DSM keeps track of location of data objects 
Multiple nodes can have read access or one node write access (multiple 
readers-one writer protocol) 
After a write, all copies are invalidated or updated 
DSM has to keep track of locations of all copies of data objects. Examples of 
implementations: 
e IVY: owner node of data object knows all nodes that have copies 
e PLUS: distributed linked-list tracks all nodes that have copies 


Advantage 
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e The read-replication can lead to substantial performance improvements 
if the ratio of reads to writes is large 


2.4. The Full-Replication Algorithm 

Extension of read-replication algorithm: multiple nodes can read and multiple 
nodes can write (multiple-readers, multiple-writers protocol) 

Issue: consistency of data for multiple writers 

Solution: use of gap-free sequencer 


. All writes sent to sequencer 

. Sequencer assigns sequence number and sends write request to 
all sites that have copies 

. Each node performs writes according to sequence numbers 

. A gap in sequence numbers indicates a missing write request: 


node asks for retransmission of missing write requests. 


Distributed shared memory 


DSM appears as 
memory in address 
space of process 


Process 
accessing DSM 


Physical 
memory 


Physical 
memory 


Physical 
memory 


NODE 1 NODE 2 


Mapping 
Manager 


Mapping 
Manager 


Shared Memory 


* When a processor O opens a scope previously 
closed by another processor P, P propagates the 
updates made within the same scope to O 


In LRC, P propagates 
both the updates of x 
and y to Q while in 
ScC, P propagates the 
update of y only since 
only y is updated in 


scope 


| Acg(LO) | 
a=x> 


—A<< the same scope as it 
Y. is read by Q. 


3. CONSISTENCY MODELS 


A consistency model is the definition of when modifications to data may be 
seen at a given processor. It defines how memory will appear to a 
programmer by placing restrictions on the values that can be returned by a 
read of a memory location. A consistency model must be well understood. It 
determines how a programmer reasons about the correctness of programs 
and determines what hardware and compiler optimizations may take place. 
e A consistency model is essentially a contract between the software and 
the memory. 
e If the software agrees to obey certain rules, the memory promises to 
work correctly. 
e If the software violates these rules, correctness of memory operation is 
no longer guaranteed. 
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Memory 


Virtual memory 


page table 


Processors 


3.1. Types of Consistency Models 
e Strict consistency 
e Sequential consistency (SC) 
e Release consistency (RC) 
e Scope consistency (ScC) 


3.1.1. Strict Consistency 
e Definition: any read to memory location x returns the value stored by the 
most recent write operation to x. 


3.1.2. Sequential Consistency 
e Definition: the result of any execution is the same as if the operations of 
all processors were executed in some sequential order, and the 
operations of each individual processor appear in this sequence in the 
order specified by its program. 
e Any valid interleaving is acceptable behavior, but all processes must see 
the same sequence of memory reference. 


3.1.3. Release Consistency 
Two types of access 
e Ordinary access: read and write 
e Synchronization access: acquire lock, release lock and barrier 


Rules: 
e Before an ordinary access to a shared variable is performed, all previous 
acquires done by the process must have completed successfully. 
e Before a release is allowed to be performed, all previous reads and 
writes done by the process must have completed. 


effect of sequential consistency. This model exploits knowledge of 
synchronization operations in order to relax memory consistency, while 
appearing to the programmer to implement sequential consistency (at least, 
under certain conditions that are beyond the scope of this book). For 
example, if the programmer uses a lock to implement a critical section, then 
a DSM system can assume that no other process may access the data items 
accessed under mutual exclusion within it. It is therefore redundant for the 
DSM system to propagate updates to these items until the process leaves the 
critical section. While items are left with ‘inconsistent’ values some of the 
time, they are not accessed at those points; the execution appears to be 
sequentially consistent. Adve and Hill (1990) describe a generalization of this 
notion called weak ordering: ‘(A DSM system) is weakly ordered with respect 
to a synchronization model if and only if it appears sequentially consistent to 
all software that obeys the synchronization model.’ Release consistency, 
which is a development of weak consistency. 
e Coherent memory: when value returned by read operation is the 
expected value (e.g., value of most recent write) 
e Mechanisms that control/synchronizes accesses is needed to maintain 
memory coherence. 


4.1. Coherence Protocol 

Concerns 
e How do we ensure that all replicas have the same information 
e How do we ensure that nodes do not access stale data 


4.1.1. Write-invalidate protocol 
A write to shared data invalidates all copies except one before write executes 
Invalidated copies are no longer accessible 
Advantage: good performance for 
e Many updates between reads 
e Per node locality of reference 


Disadvantage 
e Invalidations sent to all nodes that have copies 
e Inefficient if many nodes access same object 


Examples: most DSM systems: IVY, Clouds, Dash, Memnet, Mermaid, and 
Mirage 


4.1.2. Write-update protocol 
e A write to shared data causes all copies to be updated (new value sent, 
instead of validation) 
e More difficult to implement 


Shared Virtual Memory Space a 


5. DISTRIBUTED PAGE BASED 


Software DSM Layer 


R(1) Invalidation 


Local 


Memories 


4. COHERENCE 


One reaction to the cost of sequential consistency is to settle for a weaker 
model with well-defined properties. Coherence is an example of a weaker 
form of consistency. Under coherence, every process agrees on the order of 
write operations to the same location, but they do not necessarily agree on 
the ordering of write operations to different locations. We can think of 
coherence as sequential consistency on a location by-location basis. 
Coherent DSM can be implemented by taking a protocol for implementing 
sequential consistency and applying it separately to each unit of replicated 
data — for example, each page. The saving comes from the fact that accesses 
to two different pages are independent and need not delay one another, 
since the Protocol is applied separately to them. Weak consistency ® Dubois 
et al. (1988) developed the weak consistency model in an attempt to avoid 
the costs of sequential consistency on multiprocessors, while retaining the 
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»~ The ‘Page Based Distributed Shared Memory 
System’ consists of a collection of clients or 
workstations connected to a server by a 
Local Area Network. The server contains a 
shared memory segment within which the 
distributed database is located. The shared 
memory segment is divided in the form of 
pages and hence the name ‘Page Based 
Distributed Shared Memory System’ where 
each page represents a table within that distributed database. In the simplest 
variant, each page is present on exactly one machine. A reference to a local 
page is done at full memory speed. An attempt to reference a page on a 
different machine causes a page fault, which is trapped by the software. The 
software then sends a message to the remote machine, which finds the 
needed page and sends it to the requesting process. The fault is then 
restarted and can now complete, which is achieved with the help of Inter 
Process Communication (IPC) library. In essence, this design is similar to 
traditional virtual memory systems: when a process touches a nonresident 
page, a fault occurs and the operating system fetches the page and maps it 
in. The difference here is that instead of getting the page from the disk, the 
software gets it from another processor over the network. To the user 
process, however, the system looks very much like a_ traditional 
multiprocessor, with multiple processes are free to read and write the shared 
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memory at will. All communication and synchronization is done via the 
memory, with no communication visible to the user process. The approach is 
not to share the entire address space, but only a selected portion of it, 
namely just those variables or data structures that needs to be used by more 
than one process. With respect to a distributed database system, and in this 
model, the shared variables represent the pages or tables within the shared 
memory segment. One does not think of each machine as having direct 
access to an ordinary memory but rather, to a collection of shared variables, 
giving a higher level of abstraction. This approach greatly reduces the 
amount of data that must be shared, but in most cases, considerable 
information about the shared data is available, such as their types, which 
helps optimize the implementation. Page-based distributed-shared memory 
takes a normal linear address space and allows the pages to migrate 
dynamically over the network on demand. Processes can access all of 
memory using normal read and write instructions and are not aware of when 
page faults or network transfers occur. Accesses to remote data are detected 
and protected by the memory management unit. In order to facilitate 
optimization, the shared variables or tables are replicated on multiple 
machines. Potentially, reads can be done locally without any network traffic, 
and writes are done using a multicopy update protocol. This protocol is 
widely used in distributed database system. The main purpose of this 
simulation is to discuss the issues in a Distributed Database System and how 
they can be overcome with the help of a Page Based Distributed Shared 
Memory System. 


6. DESIGN ISSUES 


Granularity: size of shared memory unit 

e lf DSM page size is a multiple of the local virtual memory (VM) 
management page size (supported by hardware), then DSM can be 
integrated with VM, i.e. use the VM page handling 

e Advantages vs. disadvantages of using a large page size: 

e (+) Exploit locality of reference 

e (+) Less overhead in page transport 

e (-) More contention for page by many processes 

e Advantages vs. disadvantages of using a small page size 

e (+) Less contention 

e (+) Less false sharing (page contains two items, not shared but needed 
by two processes) 

e (-) More page traffic 

e Examples 

e PLUS: page size 4 Kbytes, unit of memory access is 32-bit word 

e Clouds, Munin: object is unit of shared data structure 


Page replacement 

e Replacement algorithm (e.g. LRU) must take into account page access 
modes: shared, private, read-only, writable 

e Example: LRU with access modes 

e Private (local) pages to be replaced before shared ones 

e Private pages swapped to disk 

e Shared pages sent over network to owner 

e Read-only pages may be discarded (owners have a copy). 


7. ADVANTAGES OF DISTRIBUTED SHARED MEMORY 
(DSM) 


e Data sharing is implicit, hiding data movement (as opposed to 
‘Send'/‘Receive’ in message passing model) 

Passing data structures containing pointers is easier (in message passing 
model data moves between different address spaces) 

Moving entire object to user takes advantage of locality difference 

Less expensive to build than tightly coupled multiprocessor system: off- 
the-shelf hardware, no expensive interface to shared physical memory 
Very large total physical memory for all nodes: Large programs can run 
more efficiently 

No serial access to common bus for shared physical memory like in 
multiprocessor systems 

Programs written for shared memory multiprocessors can be run on 
DSM systems with minimum changes 
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8. DISADVANTAGES OF DISTRIBUTED SHARED 
MEMORY (DSM) 


e May incur a performance penalty. 

e Must provide for protection against simultaneous access to shared data 
(locks, etc.). 

e Little programmer control over actual messages being generated. 

e Performance of irregular problems in particular may be difficult. 


9. METHODS OF ACHIEVING DSM 


Hardware Implementation 

Providing hardware support to DSM has been exploited in several ways. 
Some systems explore the idea of enhancing the networking capabilities of 
the loosely-coupledsystems. For example, the development of MemNet is 
based on the observation that the network is always treated as an I/O device 
by the communication protocols. MemNet is a shared memory local area 
network, based on a high-speed token ring, where the local network appears 
as memory in the physical address space of each processor. Capnet extends 
this idea to a wider domain, namely wide area networks, whereas the DASH 
system aims at building a scalable high performance architecture with single 
address space and coherent caches. 


Software Implementation 
During the past decade, several prototypes have been built that provide a 
DSM abstraction at the system level. System level implementations usually 
integrate the DSM as a region of virtual address space in the participating 
programs using the virtual memory management system of the underlying 
operating system. Li’s shared virtual memory provides users with an interface 
similar to the memory address space on a multiprocessor architecture. Later, 
he expanded this idea to other architectures and developed a prototype 
called Shiva on a hypercube. 

e Page based - Using the system's virtual memory 

e Shared variable approach- Using routines to access 

e shared variables 

e Object based- Shared data within collection of objects. 

e Access to shared data through object oriented discipline. 


10. EXAMPLES OF DSM 

Kerrighed - 
Kerrighed provides several features such as adistributed shared 
memory with a sequential consistency model, processes migration from one 
cluster node to another, and to a limited extent check pointing. Kerrighed 
introduces a container concept: This entity is an abstraction of both files and 
memory. 


OpenSSI 

OpenSSI is an open source single-system image clustering system. It allows a 
collection of computers to be treated as one large system, allowing 
applications running on any one machine access to the resources of all the 
machines in the cluster. 


MOSIX 

MOSIX is a distributed operating system.. In a MOSIX cluster/grid there is no 
need to modify or to link applications with any library, to copy files or login 
to remote nodes, or even to assign processes to different nodes - it is all 
done automatically, like in an SMP. 


11. CONCLUSION 

Distributed Shared Memory (DSM), in Computer Architecture is a form of 
memory architecture where the (physically separate) memories can be 
addressed as one (logically shared) address space. Here, the 
term shared does not mean that there is a single centralized memory 
but shared essentially means that the address space is shared (same physical 
address on two processors refers to the same _ location in 
memory). Distributed Global Address Space (DGAS), is a similar term for a 
wide class of software and hardware implementations, in which each node of 
a cluster has access to shared memory in addition to each node's non-shared 
private memory. Software DSM systems can be implemented in an operating 
system (OS), or as a programming library and can be thought of as 
extensions of the underlying virtual memory architecture. When 
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implemented in the OS, such systems are transparent to the developer; which 
means that the underlying distributed memory is completely hidden from 
the users. In contrast, Software DSM systems implemented at the library or 
language level are not transparent and developers usually have to program 
differently. However, these systems offer a more portable approach to DSM 
system implementation. Software DSM systems also have the flexibility to 
organize the shared memory region in different ways. The page based 
approach organizes shared memory into pages of fixed size. In contrast, the 
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object based approach organizes the shared memory region as an abstract 
space for storing shareable objects of variable sizes. Another commonly seen 
implementation uses a tuple space, in which the unit of sharing is a tuple. 
Shared memory architecture may involve separating memory into shared 
parts distributed amongst nodes and main memory; or distributing all 
memory between nodes. A coherence protocol, chosen in accordance with 
a consistency model, maintains memory coherence. 


1. This work, within the limit of available resource, has provided useful information about the distributed shared memory and its related problems. 
2. It has availed researchers the opportunity to research more on the usefulness of shared memory in distributed operating systems. 


FUTURE ISSUES 


From understanding the concept of Distributed shared memory and its implementations, one can easily implement the same making it easy for large programs 
to be stored in memory and run efficiently. Future issues will focus on further improving the performance of DSM's. 
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